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ABSTRACT 



This paper examines two common criticisms of the method that 
"U.S. News and World Report” used in its annual rankings of the academic 
quality of colleges and graduate schools. There are constant changes to the 
formula that make it impossible to interpret yearly shifts in a school's 
relative ranking in terms of change in its academic quality and the overly 
precise nature of the scores used to rank schools. This paper does not judge 
the overall concept of academic quality rankings and the validity of 
different indicators; it focuses on the extent to which methodological issues 
may affect the interpretation of the "U.S. News" rankings. To gauge the 
changes in ranking formulas, four types of change were identified and tracked 
over 6 years: (1) changes in the weight assigned to an indicator; (2) the 
removal of an indicator; (3) the addition of an indicator; and (4) changes in 
an indicator's definition or methodology. Five graduate discipline rankings 
and two undergraduate rankings (national university and national liberal 
arts) were examined. Analyses show that it is generally not possible to 
interpret year-to-year shifts in a school's rank in terms of its academic 
quality. It is possible, depending on the ranking, to make cross-year 
comparisons of a school's relative performance on between one -third and 
three-quarters of the individual indicators used. Results of the error 
analyses call into question the use of overall scores to assign schools to 
individual ranks. Recommendations are made for improving the interpretability 
and usefulness of the "U.S. News” rankings. Current "U.S. News" College and 
Graduate School Ranking Methodology is appended. (Contains 15 tables and 15 
references.) ( SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



LiZ aa 098ZE0W1 



OMVr'P f D ^ PAR ™ ( ENT 0F EDUCATION 
Office of Educational Research and Improvement 

J 'UCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



r 



PERMISSION TO REPRODUCE AND 
disseminate this material has 

BEEN GRANTED BY 



T ° l Mcno?,V? ATI0NAL RESOURCES 
.j INFORMATION CENTER (ERIC) 



Quantifying Quality: Technical Issues Surrounding U.S. News and World Report 
Rankings of U.S. Colleges and Graduate Schools 



By 

Marguerite Clarke 
CSTEEP 332F 
Lynch School of Education 
Boston College 
Chestnut Hill, MA 02467 
Email: clarkemd@hc.edu 
Telephone: 617-552-0665 



Paper Presented at the 2001 AERA Annual Meeting in Seattle 




BEST COPY AVAILABLE 



2 



M. Clarke, AERA 2001 Paper Presentation 



Introduction 

Every year, U.S. News and World Report’s (U.S. News) rankings of the academic 
quality of colleges and graduate schools hit the newsstands 1 . Their arrival brings delight 
to some and dismay to others, depending on whether their institution rose or fell in the 
quality ratings. An improved ranking can lead to increased donations from proud alumni 
and more and better qualified students in next year’s applicant pool (Monks and 
Enhrenberg, 1999). A fall can lead to tighter alignment of institutional benchmarks and 
goals with ranking criteria and pressure on admissions staff to bring in “better” applicants 
(Mufson, 1999). All the while, a question goes unanswered: What do these rankings 
really tell us about the quality of higher education? 

As a step toward answering this question, this paper examines two common 
criticisms of the methodology that U.S. News uses to rank colleges and graduate schools. 
These are: (1) constant changes to the formula make it impossible to interpret yearly 
shifts in a school’s rank in terms of change in its relative academic quality (Levin, 1999; 
Pellegrini, 1999), and (2) the score used to assign schools to ranks is overly precise, 
creating a vertical column where a group might more properly exist (Machung, 1998; 
Smetanka, 1998). The first section of this paper gives a brief introduction to the U.S. 
News rankings as well as the questions addressed by this study. The next section outlines 
the methodology used to answer these questions and the results of the analyses. The final 
section presents conclusions and recommendations. 

Before proceeding, a caveat is in order. While many have questioned the overall 
concept of academic quality rankings as well as the validity of the different indicators and 
weights used, the author of this paper suspends judgment on those issues to focus on the 
extent to which methodological issues may impact the interpretation of the U.S. News 
rankings. 



1 The term “rankings,” as used in this paper, refers to a list of schools or universities that are ordered 
according to their overall score on a formula created by U.S. News. Thus, the business rankings are a list of 
business schools ordered according to their overall score on a formula that U.S. News uses to rank graduate 
schools of business, and the national university rankings are a list of schools ordered according to their 
overall score on a formula that U.S. News uses to rank national universities. The year appended to a 
ranking is the calendar year in which it was released - i.e., the 2000 education rankings were published in 
the year 2000. 
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Background on the U.S. News Rankings 

U.S. News published its first rankings of the academic quality of colleges in 1983, 
the same year that the National Commission on Excellence in Education released A 
Nation at Risk, its influential report blasting the quality of education in America. Based 
on a survey of college presidents, the magazine listed Stanford, Harvard, and Yale as the 
top three national universities and Amherst, Swarthmore, and Williams as the top three 
national liberal arts colleges. By 1987, U.S. News had moved to a multidimensional 
approach, weighting and combining information on faculty accomplishments, student 
achievements, and institutional academic resources to produce an overall score on which 
to rank colleges. Rankings of graduate schools of business, engineering, law, and 
medicine/primary-care also appeared in this year and used a similar weight-and-sum 
approach (rankings of graduate schools of education did not appear until 1994). 

The most recent rankings still use this basic approach. At the undergraduate 
level, schools are categorized by mission and region (e.g., national universities, national 
liberal arts colleges, regional universities, and regional liberal arts colleges). Up to 
sixteen pieces of information are collected on schools in each category, including 
academic reputation; freshmen retention and graduation rates; average test scores for 
entering students; per-student spending; and alumni-giving rate. These indicators are 
standardized, weighted, and summed to produce an overall score on which to rank 
schools in each category against their peers. 

At the graduate level, schools are categorized by type - business, education, 
engineering, law, and medicine/primary-care. Depending on the type of school, data on 
up to fourteen indicators - including test scores, research expenditures, graduate 
employment rates, and reputation - are collected. Similar to the undergraduate rankings, 
the indicators are standardized, weighted, and summed to produce an overall score on 
which to rank schools in each category against their peers. Detailed information on the 
indicators and methodology that U.S. News uses to rank colleges and graduate schools is 
found in Appendix A. 2 



2 It is worth noting that several of the indicators - such as test scores, reputation, research expenditure, and 
faculty awards - have been used traditionally to measure quality (Hattendorf, 1993: Webster, 1986). 
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